Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Create physionet-build Github CI action #1320

Merged
merged 13 commits into from
Apr 21, 2021
Merged

Create physionet-build Github CI action #1320

merged 13 commits into from
Apr 21, 2021

Conversation

briangow
Copy link
Contributor

@briangow briangow commented Apr 14, 2021

[WIP] This github action will replace the shippable.yml for build-test.

This action is my initial attempt to replicate the process that has been used in the shippable.yml with Github actions. Please do not merge it yet.

I've added descriptions for what the code is doing in the -name: fields. If you could review these and suggest corrections it would be appreciated.

Regarding the lines of code starting with coverage. This code is designed to report on how much of the code was actually executed: https://coverage.readthedocs.io/en/coverage-5.5/ . Do we want to keep doing these coverage checks? If so, is it sufficient to report the result in the log instead of saving it to a file as was done in shippable.yml?

[WIP] This github action will replace the shippable.yml for build-test.
@Lucas-Mc
Copy link
Contributor

Lucas-Mc commented Apr 14, 2021

Exit with code 127 source: not found ... maybe . env/bin/activate since source isn't part of /bin/sh

EDIT
Here's another way:

defaults:
  run:
    shell: bash

@tompollard
Copy link
Member

You should be able to fix the postgres error by adding postgres as a service:

    services:
      postgres:
        image: postgres:11
        env:
          POSTGRES_DB: physionet        
          POSTGRES_PASSWORD: password
          POSTGRES_USER: physionet
        ports:
          - 5432:5432
        # Set health checks to wait until postgres has started
        options: >-
          --health-cmd pg_isready
          --health-interval 10s
          --health-timeout 5s
          --health-retries 5

The manual installation can then be removed:

  • remove apt-get install postgresql
  • remove service postgresql start
  • remove the setup postgres chunk

@bemoody
Copy link
Collaborator

bemoody commented Apr 15, 2021 via email

@tompollard
Copy link
Member

I think that is not a good idea. (a), we want to test against the
Debian stable version of postgres, not whatever version is supplied by
github.

Yep, fair point!

@Lucas-Mc
Copy link
Contributor

Lucas-Mc commented Apr 15, 2021 via email

@tompollard
Copy link
Member

What other options do we have?

I think we go back to the original approach, which is to install and start a postgres service on the server.

@bemoody
Copy link
Collaborator

bemoody commented Apr 15, 2021 via email

@tompollard
Copy link
Member

Brian, you said in ad7dead that Docker runs as root; I'm not sure if that's true or not, but either way, there are some commands (like installing lightwave) that require root privileges, while other commands (e.g. manage.py test) specifically should be run as a non-root user so that we know the permission checks work correctly.

In this case, we probably need to add a step at the beginning of the process to create some kind of build user. Seemed strange to be running everything as root anyway.

@briangow
Copy link
Contributor Author

I'll look into setting up a separate user from root. @bemoody , do we need to store a file for the logs?

@bemoody
Copy link
Collaborator

bemoody commented Apr 15, 2021 via email

@briangow
Copy link
Contributor Author

@bemoody I confirmed with whoami in the Docker container that it is running as root. This indicates that we should only be running as root when using a Docker container inside Github actions:

"USER
Docker actions must be run by the default Docker user (root). Do not use the USER instruction in your Dockerfile, because you won't be able to access the GITHUB_WORKSPACE "
https://docs.github.com/en/actions/creating-actions/dockerfile-support-for-github-actions

@bemoody
Copy link
Collaborator

bemoody commented Apr 16, 2021 via email

@briangow
Copy link
Contributor Author

Yeah, I'd also seen that. I wasn't able to figure out how to create a non-root user before issuing the --user command and have the created user be seen at the options: --user line. I thought this would be helpful since they also needed to do a "pre-step" before calling their Docker container setup: actions/runner#812 . However, I haven't been able to successfully setup a Docker container under the steps: section.

This gives a bit of information about GITHUB_WORKSPACE:

GITHUB_WORKSPACE | The GitHub workspace directory path. The workspace directory is a copy of your repository if your workflow uses the actions/checkout action. If you don't use the actions/checkout action, the directory will be empty. For example, /home/runner/work/my-repo-name/my-repo-name.
https://docs.github.com/en/actions/reference/environment-variables

I can look into other options for checking out the repository without using the checkout action.

@briangow
Copy link
Contributor Author

I can run a given command (whoami in this case) as a non-root user with:
sudo -u physionet_user whoami

Also, I realized that the choice of --user 1001 from this thread: https://github.community/t/how-to-run-action-as-a-non-root-user/17572/2 , wasn't a random choice as /home/1001 already exists. However, since this account doesn't have privileges I can't do much with it after issuing --user 1001.

@bemoody
Copy link
Collaborator

bemoody commented Apr 16, 2021 via email

@briangow
Copy link
Contributor Author

briangow commented Apr 17, 2021

Thanks @bemoody , your suggestion does work for setting up a pntestuser and switching to that user for a given run:

I'm still trying to work out if it's possible to clone physionet-build to a location that is not under $GITHUB_WORKSPACE since we can only access that workspace as root (which I confirmed). Instead of using the github action checkout@v2, I installed git and ran:

      - name: clone physionet-build
        run: |
          git clone https://github.com/MIT-LCP/physionet-build.git

running ls -la $GITHUB_WORKSPACE
gives:

total 12
drwxr-xr-x 3 pntestuser pntestuser 4096 Apr 17 02:28 .
drwxr-xr-x 3       1001        121 4096 Apr 17 02:27 ..
lrwxrwxrwx 1 root       root         12 Apr 17 02:28 .env -> .env.example
drwxr-xr-x 7 root       root       4096 Apr 17 02:28 physionet-build

@briangow
Copy link
Contributor Author

@bemoody , from what I can tell Github actions doesn't allow access to cloned repositories (in the GITHUB_WORKSPACE) while running as a non-root user. This post explains some of the restrictions around not being able to clone outside of the GITHUB_WORKSPACE:

"Today the path input is the only way to change the directory where the code is cloned. Also there is a restriction that it must be under GITHUB_WORKSPACE."
actions/checkout#197

As mentioned previously, you can't access the GITHUB_WORKSPACE unless you are root. A workaround would be to use a self-hosted runner but I don't think we want to do that. I could ask in a forum to check to see if there is a way around this that I haven't been able to find so far.

@bemoody
Copy link
Collaborator

bemoody commented Apr 20, 2021 via email

@tompollard
Copy link
Member

tompollard commented Apr 20, 2021

As this is currently just supplementing existing tests, I'd be good to merge now if everyone else is okay with that? We can then add new issues for (1) fixing the user issue and (2) triggering rebuilds on pull requests when the target is updated.

@Lucas-Mc
Copy link
Contributor

It looks like neither these nor the shippable tests are run when the dev branch is updated, so the final result isn't being tested until after it's merged.

Can you explain more of what you mean by that? This:

on:
   push:
     branches:
       - dev
   pull_request:
     branches:
       - dev

should be triggering tests whenever a commit is pushed to dev or a pull request is targeting dev (new commits to the pull request also trigger tests).

Are you saying we require all checks to pass before it can be merged? I think this is just a protected branch issue and not GHA issue.

@bemoody
Copy link
Collaborator

bemoody commented Apr 21, 2021 via email

@bemoody bemoody merged commit 2ceb075 into dev Apr 21, 2021
@bemoody
Copy link
Collaborator

bemoody commented Apr 21, 2021 via email

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants